requirements.txt (1)

Warning: Dependencies still in use but removed from requirements

The removed packages crawlee[playwright] and defusedxml are still actively used in the codebase:

• dags/hivemind_etl_helpers/src/db/website/crawlee_client.py depends on both removed packages:
  • Uses from crawlee.playwright_crawler import PlaywrightCrawler
  • Uses from defusedxml import ElementTree for XML sitemap parsing
• The client is actively used in the ETL pipeline as shown in test_website_etl.py

These dependencies should be restored to requirements.txt or the crawlee_client.py implementation should be updated to use alternative packages.

🔗 Analysis chain

Line range hint 1-24: Verify the impact of removed dependencies

The removal of crawlee[playwright] and defusedxml suggests that web scraping and XML parsing functionalities have been moved or are now handled differently.

Let's verify if any remaining code still depends on these packages:

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Check for any remaining usage of removed packages

# Check for crawlee imports or usage
rg -l "from crawlee|import crawlee"

# Check for defusedxml imports or usage
rg -l "from defusedxml|import defusedxml"

# Check for XML parsing related files
fd -e xml

Length of output: 211

---

Script:

#!/bin/bash
# Let's check the content of crawlee_client.py to understand the usage
cat dags/hivemind_etl_helpers/src/db/website/crawlee_client.py

# Also check if there are any other files importing or using this client
rg -l "crawlee_client"

Length of output: 3918

---

Script:

#!/bin/bash
# Let's check the test file to see if the client is still being used
cat dags/hivemind_etl_helpers/tests/unit/test_website_etl.py

# Also check for any other files that might be using XML parsing
rg -l "ElementTree|\.xml|fromstring"

Length of output: 2728

dags/hivemind_etl_helpers/src/db/discord/find_guild_id.py (1)

Line range hint 5-24: Consider enhancing function documentation.

While the function implementation is solid, the docstring could be more comprehensive:

• Add return type documentation
• Document the possible ValueError exception

Here's the suggested improvement:

 def find_guild_id_by_platform_id(platform_id: str) -> str:
     """
     find the guild id using the given platform id
 
     Parameters
     ------------
     platform_id : str
         the community id that the guild is for
+
+    Returns
+    -------
+    str
+        The Discord guild ID associated with the platform
+
+    Raises
+    ------
+    ValueError
+        If the platform_id does not exist or is not associated with Discord
     """

dags/analyzer_helper/discord/discord_load_transformed_members.py (1)

Line range hint 16-20: Add error handling and data validation

The load method performs critical database operations without proper error handling or data validation. Consider these improvements:

1. Add try-catch blocks around MongoDB operations
2. Validate processed_data structure before insertion
3. Consider chunking for large datasets to prevent memory issues

def load(self, processed_data: list[dict], recompute: bool = False):
+    if not processed_data:
+        logging.warning("No data to load")
+        return
+
+    try:
        if recompute:
            logging.info("Recompute is true, deleting all the previous data!")
            self.collection.delete_many({})
-        self.collection.insert_many(processed_data)
+        
+        # Insert in chunks to handle large datasets
+        chunk_size = 1000
+        for i in range(0, len(processed_data), chunk_size):
+            chunk = processed_data[i:i + chunk_size]
+            self.collection.insert_many(chunk)
+            logging.info(f"Inserted {len(chunk)} records")
+    except Exception as e:
+        logging.error(f"Failed to load data: {str(e)}")
+        raise

dags/hivemind_etl_helpers/mediawiki_etl.py (1)

Line range hint 39-42: Consider making the collection name configurable

The collection name is currently hardcoded as "mediawiki". Consider making it configurable through a parameter or configuration to improve flexibility and reusability.

 def process_mediawiki_etl(
     community_id: str,
     api_url: str,
     page_titles: list[str],
+    collection_name: str = "mediawiki",
 ) -> None:
     # ...
     ingestion_pipeline = CustomIngestionPipeline(
         community_id=community_id,
-        collection_name="mediawiki"
+        collection_name=collection_name
     )

dags/hivemind_etl_helpers/tests/integration/test_ingestion_pipeline_etl.py (1)

Line range hint 10-41: Review test coverage for the new implementation

While the basic functionality is tested, consider enhancing the test coverage:

1. The test_run_pipeline method only tests the happy path with two documents
2. Missing tests for:
   • Empty document list
   • Invalid document format
   • Different community IDs and collection names
   • Non-testing mode behavior

Would you like me to help generate additional test cases to improve coverage?

dags/hivemind_google_drive_etl.py (2)

Line range hint 44-47: Consider making the collection name configurable

The collection name "google" is hardcoded. Consider making it configurable through environment variables or DAG configuration for better flexibility and maintainability.

         ingest_data = CustomIngestionPipeline(
-            community_id=community_id, collection_name="google"
+            community_id=community_id, collection_name=Variable.get("GDRIVE_COLLECTION_NAME", "google")
         )

---

Security concerns identified in refresh token handling

Based on the code analysis, there are several security issues that need to be addressed:

1. The refresh token is being retrieved from community_information dictionary without any error handling for missing or invalid tokens
2. The refresh token is stored in environment variables instead of using Airflow's secret backend
3. No token expiration or validation checks are implemented in the GoogleDriveLoader class
4. No error handling exists for token invalidation scenarios in the load_data method

Recommendations:

• Move token storage to Airflow's Variables or Connections for secure credential management
• Add error handling for token expiration/invalidation in GoogleDriveLoader
• Implement token validation before usage
• Add try-except blocks around token operations with appropriate error messages that don't expose the token

🔗 Analysis chain

Line range hint 35-41: Review refresh token handling security

The refresh token is being passed directly to the GoogleDriveLoader. Ensure that:

1. The token is stored securely in Airflow's secret backend
2. Proper error handling exists for token expiration/invalidation
3. The token is not logged in any error messages

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Check for potential token exposure in logs and error handling

# Check for any logging of refresh tokens
rg -B 2 -A 2 "refresh_token.*log|log.*refresh_token" --type python

# Check for error handling around token usage
ast-grep --pattern 'try {
  $$$
  refresh_token
  $$$
} catch ($$$) {
  $$$
}'

Length of output: 180

---

Script:

#!/bin/bash
# Search for refresh token usage and error handling with correct file type
rg -t py -B 2 -A 2 "refresh_token.*log|log.*refresh_token"

# Look for try-except blocks around refresh token usage
rg -t py -B 3 -A 3 "refresh_token" 

# Check GoogleDriveLoader implementation for error handling
fd "google.*drive.*loader" -t f -x cat {}

# Check for secret management patterns
rg -t py "Variable.get|secret" -B 2 -A 2

Length of output: 26562

dags/hivemind_etl_helpers/src/utils/modules/discourse.py (1)

Line range hint 31-53: Consider simplifying nested loops with list comprehension.

While the current implementation is correct, the nested loops and conditions could be made more Pythonic using list comprehension.

Here's a suggested refactoring that maintains the same logic but improves readability:

         platforms_data: list[dict[str, str | datetime]] = []
-        # for each community module
-        for module in modules:
-            community = module["community"]
-
-            # each platform of the community
-            for platform in module["options"]["platforms"]:
-                if platform["name"] != self.platform_name:
-                    continue
-
-                # learning is for doing ETL on data
-                if "learning" in platform["metadata"]:
-                    learning_config = platform["metadata"]["learning"]
-
-                    platforms_data.append(
-                        {
-                            "community_id": str(community),
-                            "endpoint": learning_config["endpoint"],
-                            "from_date": learning_config["fromDate"],
-                        }
-                    )
+        platforms_data.extend([
+            {
+                "community_id": str(module["community"]),
+                "endpoint": platform["metadata"]["learning"]["endpoint"],
+                "from_date": platform["metadata"]["learning"]["fromDate"],
+            }
+            for module in modules
+            for platform in module["options"]["platforms"]
+            if platform["name"] == self.platform_name
+            and "learning" in platform["metadata"]
+        ])

dags/hivemind_etl_helpers/src/db/telegram/utils/platform.py (1)

Line range hint 8-85: Consider enhancing error handling and type hints

While the implementation is functionally correct, here are some suggestions for improvement:

1. Add error handling for MongoDB operations:

     def check_platform_existence(self) -> tuple[ObjectId | None, ObjectId | None]:
-        document = self._client[self.database][self.collection].find_one(
-            {"metadata.id": self.chat_id},
-            {
-                "community": 1,
-                "_id": 1,
-            },
-        )
+        try:
+            document = self._client[self.database][self.collection].find_one(
+                {"metadata.id": self.chat_id},
+                {
+                    "community": 1,
+                    "_id": 1,
+                },
+            )
+        except Exception as e:
+            logger.error(f"Failed to check platform existence: {e}")
+            raise

2. Use class constants for database and collection names:

     def __init__(self, chat_id: int, chat_name: str) -> None:
+        DATABASE_NAME = "Core"
+        COLLECTION_NAME = "platforms"
         self._client = MongoSingleton.get_instance().get_client()
         self.chat_id = chat_id
         self.chat_name = chat_name
-        self.database = "Core"
-        self.collection = "platforms"
+        self.database = DATABASE_NAME
+        self.collection = COLLECTION_NAME

3. Add more specific type hints:

-    def check_platform_existence(self) -> tuple[ObjectId | None, ObjectId | None]:
+    def check_platform_existence(self) -> tuple[ObjectId | None, ObjectId | None]:
+        """
+        Returns:
+            tuple[ObjectId | None, ObjectId | None]: A tuple containing (community_id, platform_id)
+        Raises:
+            PyMongoError: If there's an error accessing the database
+        """

dags/hivemind_etl_helpers/src/utils/modules/github.py (1)

Line range hint 31-39: Docstring needs to be updated to reflect all actual return fields

The docstring example is incomplete as it doesn't show the repo_ids field which appears in the implementation (though commented out).

Consider updating the example to show all possible fields:

             [{
                 "community_id": "community1",
                 "organization_ids": ["1111", "2222"],
+                "repo_ids": ["132", "45232"],
                 "from_date": None
             }]

dags/hivemind_etl_helpers/github_etl.py (2)

Line range hint 16-77: Consider adding error handling and performance optimizations.

The ETL pipeline processes multiple GitHub data types in memory. Consider the following improvements:

1. Error Handling:

   • Add error handling for API rate limits
   • Handle potential memory issues with large datasets
   • Add retries for network operations

2. Performance:

   • Consider processing data in batches
   • Add progress logging for long-running operations
   • Implement parallel processing for independent operations

Here's a suggested improvement for error handling and batching:

 def process_github_vectorstore(
     community_id: str,
     github_org_ids: list[str],
     repo_ids: list[str],
     from_starting_date: datetime | None = None,
+    batch_size: int = 1000,
+    max_retries: int = 3,
 ) -> None:
     """
     ETL process for github raw data
     ...
     """
     load_dotenv()
     prefix = f"COMMUNITYID: {community_id} "
     logging.info(f"{prefix}Processing data!")

+    try:
         org_repository_ids = get_github_organization_repos(
             github_organization_ids=github_org_ids
         )
         repository_ids = list(set(repo_ids + org_repository_ids))
         logging.info(f"{len(repository_ids)} repositories to fetch data from!")

         # EXTRACT with retries
+        for attempt in range(max_retries):
+            try:
                 github_extractor = GithubExtraction()
                 github_comments = github_extractor.fetch_comments(repository_id=repository_ids)
                 github_commits = github_extractor.fetch_commits(repository_id=repository_ids)
                 github_issues = fetch_issues(repository_id=repository_ids)
                 github_prs = fetch_pull_requests(repository_id=repository_ids)
+                break
+            except Exception as e:
+                if attempt == max_retries - 1:
+                    raise
+                logging.warning(f"Attempt {attempt + 1} failed: {str(e)}")

         # Process in batches
+        for i in range(0, len(all_documents), batch_size):
+            batch = all_documents[i:i + batch_size]
+            logging.info(f"Processing batch {i//batch_size + 1}")
             ingestion_pipeline = CustomIngestionPipeline(community_id, collection_name="github")
-            ingestion_pipeline.run_pipeline(docs=all_documents)
+            ingestion_pipeline.run_pipeline(docs=batch)
+    except Exception as e:
+        logging.error(f"Failed to process GitHub data: {str(e)}")
+        raise

---

Line range hint 16-77: Add security considerations for sensitive data handling.

The pipeline processes GitHub organization and repository data which might contain sensitive information. Consider:

1. Adding data sanitization before storage
2. Implementing access control checks
3. Adding audit logging for sensitive operations
4. Implementing data retention policies

dags/hivemind_etl_helpers/notion_etl.py (1)

3-4: Consider consolidating package dependencies

The code currently mixes imports from both tc_hivemind_backend and hivemind_etl_helpers. While this might be intentional during migration, it could lead to maintenance challenges:

• NotionExtractor is still from hivemind_etl_helpers
• CustomIngestionPipeline is from tc_hivemind_backend

Consider either:

1. Moving NotionExtractor to tc_hivemind_backend for consistency
2. Creating a migration plan to track and complete the transition of all components

dags/analyzer_helper/discourse/extract_raw_members.py (1)

Line range hint 1-100: Consider separating database concerns

The class currently manages connections to both MongoDB and Neo4j. This tight coupling to multiple databases could make the code harder to maintain and test. Consider:

1. Extracting the database operations into separate repository classes
2. Using dependency injection for database clients
3. Implementing a unit of work pattern for managing multiple data sources

Would you like me to propose a refactored structure that better separates these concerns?

dags/analyzer_helper/telegram/extract_raw_members.py (2)

Line range hint 15-17: Add error handling for MongoDB operations

The MongoDB client initialization and collection access lack error handling. Consider adding try-catch blocks to handle potential connection issues and implementing proper cleanup.

 def __init__(self, chat_id: int, platform_id: str):
     """
     Initialize the ExtractRawMembers with the Neo4j connection parameters.
     """
     self.neo4jConnection = Neo4jConnection()
     self.driver = self.neo4jConnection.connect_neo4j()
     self.converter = DateTimeFormatConverter()
     self.chat_id = chat_id
-    self.client = MongoSingleton.get_instance().client
-    self.platform_db = self.client[platform_id]
-    self.rawmembers_collection = self.platform_db["rawmembers"]
+    try:
+        self.client = MongoSingleton.get_instance().client
+        self.platform_db = self.client[platform_id]
+        self.rawmembers_collection = self.platform_db["rawmembers"]
+    except Exception as e:
+        self.close()  # Ensure Neo4j connection is closed if MongoDB fails
+        raise RuntimeError(f"Failed to initialize MongoDB connection: {str(e)}")

---

Line range hint 23-26: Consider improving resource cleanup and separation of concerns

The close() method only handles Neo4j cleanup, but MongoDB connections should also be properly managed. Additionally, consider separating the Neo4j and MongoDB operations into distinct classes following the Single Responsibility Principle.

Consider refactoring into separate data access classes:

class TelegramMemberNeo4jRepository:
    # Neo4j specific operations
    pass

class TelegramMemberMongoRepository:
    # MongoDB specific operations
    pass

class ExtractRawMembers:
    def __init__(self, chat_id: int, platform_id: str):
        self.neo4j_repo = TelegramMemberNeo4jRepository()
        self.mongo_repo = TelegramMemberMongoRepository(platform_id)

dags/hivemind_etl_helpers/tests/integration/test_telegram_comminity.py (1)

Line range hint 11-24: Consider using a more distinctive test database name.

The test setup uses generic database and collection names (TempPlatforms, TempCore). While these are dropped after tests, using more specific names (e.g., prefixed with test_telegram_) would reduce the risk of conflicts in parallel test runs.

-        self.telegram_platform.collection = "TempCore"
-        self.telegram_platform.database = "TempPlatforms"
+        self.telegram_platform.collection = "test_telegram_core"
+        self.telegram_platform.database = "test_telegram_platforms"

dags/hivemind_etl_helpers/tests/integration/test_get_discourse_community_data.py (1)

Line range hint 11-15: Consider adding edge case tests

While the current test coverage is good, consider adding tests for these scenarios:

• Invalid/malformed platform metadata
• Missing optional fields
• Multiple communities with multiple platforms

Would you like me to provide example test cases for these scenarios?

Also applies to: 46-106

dags/analyzer_helper/tests/integration/test_discord_load_transformed_data.py (1)

Line range hint 11-91: Consider using a mock database for integration tests.

While the tests are well-structured and comprehensive, they interact directly with MongoDB. Consider these improvements for more robust testing:

1. Use a mock database or in-memory MongoDB for testing
2. Add environment variable checks to prevent accidental runs against production
3. Use a more distinctly named test database (e.g., "test_discord_platform")

Example implementation:

import os
from unittest.mock import patch

class TestDiscordLoadTransformedData(unittest.TestCase):
    @classmethod
    def setUpClass(cls):
        # Ensure we're in a test environment
        if not os.getenv('TESTING'):
            raise EnvironmentError("Tests must be run with TESTING=true")
        
    def setUp(self):
        self.client = MongoSingleton.get_instance().client
        self.db = self.client["test_discord_platform"]  # Clearly marked as test DB
        self.collection = self.db["rawmemberactivities"]
        self.collection.delete_many({})
        self.loader = DiscordLoadTransformedData("test_discord_platform")

dags/hivemind_etl_helpers/tests/integration/test_discord_prepare_thread_summaries.py (2)

Line range hint 23-91: Consider adding error handling test cases

The test coverage is good for happy paths, but with the migration to a new backend library, consider adding test cases for:

• MongoDB connection failures
• Invalid guild IDs
• Malformed message data

Would you like me to help generate these additional test cases?

---

Migration to tc_hivemind_backend.db.mongo is incomplete

There are still 2 files using the old hivemind_etl_helpers.src.utils.mongo import in the violation detection helpers:

• dags/violation_detection_helpers/tests/unit/test_extract_raw_data.py
• dags/violation_detection_helpers/tests/unit/test_extract_raw_data_latest_date.py

🔗 Analysis chain

Line range hint 1-91: Verify impact of backend library migration

While the changes in this file are minimal and well-implemented, this is part of a larger migration to use the hivemind backend library.

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Check for any remaining references to the old mongo utility
rg "hivemind_etl_helpers.src.utils.mongo" 

# Check for consistent usage of the new backend library
rg "tc_hivemind_backend.db.mongo"

Length of output: 7245

dags/hivemind_etl_helpers/src/db/discord/fetch_raw_messages.py (1)

Line range hint 47-143: Standardize MongoDB client access pattern

There's an inconsistency in how the MongoDB client is accessed:

• fetch_raw_messages uses get_client()
• fetch_raw_msg_grouped accesses .client directly

Consider standardizing the access pattern across all functions.

-    client = MongoSingleton.get_instance().client
+    client = MongoSingleton.get_instance().get_client()

dags/analyzer_helper/discord/fetch_discord_platforms.py (2)

Line range hint 89-89: Consider implementing the TODO suggestion to merge methods.

The TODO comment about merging fetch_all and fetch_analyzer_parameters is worth addressing. Consider refactoring these methods into a single flexible method that can handle both use cases.

Here's a suggested approach:

def fetch_platforms(self, platform_id: str = None, include_details: bool = False) -> Union[List[dict], dict]:
    """
    Fetches Discord platform(s) with configurable detail level.
    
    Args:
        platform_id (str, optional): If provided, fetches a single platform
        include_details (bool): Whether to include additional metadata
    
    Returns:
        Union[List[dict], dict]: Platform data either as a list or single dict
    """
    base_query = {
        "disconnectedAt": None,
        "name": "discord",
    }
    base_projection = {
        "_id": 1,
        "metadata.period": 1,
        "metadata.id": 1,
    }
    
    if include_details:
        base_projection.update({
            "metadata.action": 1,
            "metadata.window": 1,
            "metadata.selectedChannels": 1,
        })
    
    if platform_id:
        base_query["_id"] = ObjectId(platform_id)
        doc = self.collection.find_one(base_query, base_projection)
        if not doc:
            raise ValueError(f"No platform with platform_id: {platform_id} is available!")
        return self._format_platform_data(doc, include_details)
    
    cursor = self.collection.find(base_query, base_projection)
    return [self._format_platform_data(doc, include_details) for doc in cursor]

---

Line range hint 85-87: Enhance error message for better debugging.

The error message could be more informative by including the query parameters used.

Consider updating the error message:

-            raise ValueError(
-                f"No platform given platform_id: {platform_id} is available!"
-            )
+            raise ValueError(
+                f"No Discord platform found with id: {platform_id}. "
+                f"Query criteria: active (disconnectedAt: None), platform: discord"
+            )

dags/hivemind_telegram_etl.py (3)

Line range hint 89-147: Consider splitting the processor task for better maintainability.

The processor task handles both message and summary processing with complex conditional logic. Consider splitting it into two separate tasks for better maintainability and clearer responsibility separation.

Example refactor:

@task
def process_messages(details: dict[str, tuple[str, str] | str]) -> None:
    """Process telegram messages."""
    # Message-specific processing logic

@task
def process_summaries(details: dict[str, tuple[str, str] | str]) -> None:
    """Process telegram summaries."""
    # Summary-specific processing logic

# In the DAG:
if dag_type == "messages":
    process_messages.expand(details=details)
else:
    process_summaries.expand(details=details)

---

Line range hint 52-71: Enhance error handling in chat_existence task.

The chat_existence task could benefit from more robust error handling, especially around platform creation.

Consider adding error handling:

 @task
 def chat_existence(chat_info: tuple[str, str]) -> dict[str, tuple[str, str] | str]:
     """Check and create community & platform for Telegram if needed."""
     chat_id, chat_name = chat_info
+    try:
         platform_utils = TelegramPlatform(chat_id=chat_id, chat_name=chat_name)
         community_id, platform_id = platform_utils.check_platform_existence()

         if community_id is None:
             logging.info(f"Platform with chat_id: {chat_id} doesn't exist. Creating one!")
             community_id, platform_id = platform_utils.create_platform()

         modules = TelegramModules(community_id, platform_id)
         modules.create()

         return {
             "chat_info": chat_info,
             "community_id": str(community_id),
         }
+    except Exception as e:
+        logging.error(f"Failed to process chat {chat_name} ({chat_id}): {str(e)}")
+        raise

---

Line range hint 148-156: Consider making the lookback period configurable.

The 30-day lookback period for messages is hardcoded. Consider making this configurable through Airflow variables or environment variables for more flexibility.

Example implementation:

+from airflow.models import Variable
+
+# In the DAG:
+MESSAGE_LOOKBACK_DAYS = int(Variable.get("telegram_message_lookback_days", 30))
+
 if latest_date and dag_type == "messages":
-    from_date = latest_date - timedelta(days=30)
+    from_date = latest_date - timedelta(days=MESSAGE_LOOKBACK_DAYS)
     logging.info(f"Started extracting from date: {from_date}!")
     messages = extractor.extract(from_date=from_date)

dags/hivemind_etl_helpers/tests/integration/test_discord_prepare_llama.py (1)

Line range hint 11-206: Consider improving test maintainability and reliability

While the test coverage is comprehensive, there are several improvements that could make the tests more maintainable and reliable:

1. Use setUp/tearDown methods for MongoDB initialization and cleanup
2. Move test data to fixtures
3. Add error handling for MongoDB connection failures
4. Consider using a mock MongoDB for faster tests

Here's a suggested refactor for the test class structure:

class TestTransformRawMsgToDocument(unittest.TestCase):
    @classmethod
    def setUpClass(cls):
        cls.client = MongoSingleton.get_instance().client
        cls.guild_id = "1234"
        
    def setUp(self):
        # Clear collections before each test
        self.client[self.guild_id].drop_collection("guildmembers")
        self.client[self.guild_id].drop_collection("roles")
        
    def tearDown(self):
        # Cleanup after each test
        self.client[self.guild_id].drop_collection("guildmembers")
        self.client[self.guild_id].drop_collection("roles")
        
    def _load_test_data(self):
        # Move test data to a separate method or fixture file
        # Current test data implementation...
        
    def test_transform_two_data(self):
        messages, expected_results = self._load_test_data()
        try:
            documents = transform_discord_raw_messages(self.guild_id, messages)
            # Current assertions...
        except Exception as e:
            self.fail(f"Test failed due to: {str(e)}")

dags/hivemind_etl_helpers/tests/integration/test_pg_vector_access_with_discord.py (2)

Line range hint 84-146: Consider making test dates more maintainable.

While the test data creation is well-structured, consider extracting the hardcoded dates (e.g., datetime(2023, 5, 1), datetime(2023, 1, 1)) into class-level constants or test configuration. This would make the tests more maintainable and easier to update.

class TestPGVectorAccess(unittest.TestCase):
+    # Test configuration
+    TEST_START_DATE = datetime(2023, 1, 1)
+    TEST_MESSAGE_DATE = datetime(2023, 5, 1)

     def _create_and_save_doc(self, table: str, guild_id: str, dbname: str):
         # ...
-        "createdDate": datetime(2023, 5, 1),
+        "createdDate": self.TEST_MESSAGE_DATE,
         # ...

---

Line range hint 148-190: Fix potential resource leak in database connections.

While the cursor is properly closed, the database connection created in setUpDB is never closed. Consider adding a tearDown method to ensure proper cleanup of database resources.

class TestPGVectorAccess(unittest.TestCase):
+    def tearDown(self):
+        if hasattr(self, 'postgres_conn'):
+            self.postgres_conn.close()

dags/analyzer_helper/tests/unit/test_unit_fetch_discord_platforms.py (3)

Line range hint 264-264: Remove debug print statements

Debug print statements should be removed from test cases.

-        print("Result: ", result)
-        print("Expected result: ", expected_result)

---

Line range hint 289-289: Fix method name typo

The method name contains a double underscore which appears to be a typo.

-    def test_fetch__analyzer_parameters_empty(self, mock_get_instance):
+    def test_fetch_analyzer_parameters_empty(self, mock_get_instance):

---

Line range hint 52-59: Consider simplifying test data

The test data contains actual Discord channel IDs. Consider using simpler, more readable mock IDs for test cases (e.g., "channel1", "channel2") to improve test maintainability and readability.

Example simplification:

     "selectedChannels": [
-        "1067517728543477920",
-        "1067512760163897514",
-        "1177090385307254844",
-        "1177728302123851846",
-        "1194381466663141519",
-        "1194381535734935602",
+        "channel1",
+        "channel2",
+        "channel3"
     ],

Also applies to: 146-153

dags/violation_detection_helpers/tests/integration/test_retrieve_modeules.py (2)

Line range hint 108-117: Fix inconsistent test assertions

There are mismatches between the test data and assertions:

1. The assertion checks for platform_id "515151515151515151515154" but the test data uses "515151515151515151515153"
2. The assertion checks for resources containing "12390" but this value isn't in the test data

Apply this fix:

-            elif module["platform_id"] == "515151515151515151515154":
-                self.assertEqual(module["platform_id"], "515151515151515151515153")
-                self.assertEqual(module["community"], "515151515151515151515154")
-                self.assertEqual(module["resources"], ["7373", "8282", "12390"])
+            elif module["platform_id"] == "515151515151515151515153":
+                self.assertEqual(module["platform_id"], "515151515151515151515153")
+                self.assertEqual(module["community"], "515151515151515151515151")
+                self.assertEqual(module["resources"], ["7373", "8282", "1"])

---

Line range hint 22-186: Consider refactoring test data setup

The test data structure could be improved for better maintainability:

1. Consider extracting common test data into class-level fixtures or helper methods to reduce duplication
2. Consider using relative dates (e.g., using timedelta from current date) instead of hardcoded dates

Example refactor:

def create_module_doc(self, community_id, platform_id, platform_name, resources, emails, from_date, to_date):
    return {
        "name": "violationDetection",
        "community": ObjectId(community_id),
        "options": {
            "platforms": [{
                "platform": ObjectId(platform_id),
                "name": platform_name,
                "metadata": {
                    "selectedResources": resources,
                    "selectedEmails": emails,
                    "fromDate": from_date,
                    "toDate": to_date
                }
            }]
        },
        "createdAt": datetime.now(),
        "updatedAt": datetime.now()
    }

dags/hivemind_etl_helpers/tests/integration/test_discord_prepare_document_from_db.py (1)

Line range hint 156-167: Consider adding test coverage for webhook-generated messages

The test data structure includes isGeneratedByWebhook field, but there's no test case covering webhook-generated messages (isGeneratedByWebhook: True). Consider adding a test case to ensure proper handling of webhook messages.

Here's a suggested test message to add:

data = {
    "type": 0,
    "author": "111",
    "content": "webhook generated message",
    "user_mentions": [],
    "role_mentions": [],
    "reactions": [],
    "replied_user": None,
    "createdDate": datetime(2023, 5, 1),
    "messageId": str(np.random.randint(1000000, 9999999)),
    "channelId": channels[0],
    "channelName": "channel1",
    "threadId": None,
    "threadName": None,
    "isGeneratedByWebhook": True,  # Test webhook message
}
messages.append(data)

# Add corresponding assertion
expected_metadata_webhook = {
    "channel": "channel1",
    "date": datetime(2023, 5, 1).strftime("%Y-%m-%d %H:%M:%S"),
    "author_username": "user1",
    "author_global_name": "user1_GlobalName",
    "thread": None,
    "is_webhook": True,
}
self.assertDictEqual(documents[4].metadata, expected_metadata_webhook)
self.assertEqual(documents[4].text, "webhook generated message")

dags/hivemind_etl_helpers/tests/integration/test_discord_prepare_grouped_data.py (2)

Line range hint 13-24: Consider improving test setup documentation and maintainability.

The setup_db method would benefit from:

1. Adding a docstring to explain the purpose and parameters
2. Moving magic values (like ObjectIds) to class-level constants
3. Consider using a test data factory pattern for cleaner test data generation

Here's a suggested improvement:

 class TestDiscordGroupedDataPreparation(TestCase):
+    # Test constants
+    COMMUNITY_ID = ObjectId("9f59dd4f38f3474accdc8f24")
+    PLATFORM_ID = ObjectId("063a2a74282db2c00fbc2428")
+    DEFAULT_GUILD_ID = "1234"
+
     def setup_db(
         self,
         channels: list[str],
         create_modules: bool = True,
         create_platform: bool = True,
-        guild_id: str = "1234",
+        guild_id: str = DEFAULT_GUILD_ID,
     ):
+        """Set up test database with required collections and documents.
+        
+        Args:
+            channels: List of Discord channel IDs to include
+            create_modules: Whether to create the modules collection
+            create_platform: Whether to create the platforms collection
+            guild_id: Discord guild ID to use for the test
+        """
         client = MongoSingleton.get_instance().client
-
-        community_id = ObjectId("9f59dd4f38f3474accdc8f24")
-        platform_id = ObjectId("063a2a74282db2c00fbc2428")

---

Line range hint 127-186: Consider reducing test data duplication.

The test data generation is repetitive across test methods. Consider creating helper methods or using a test data factory pattern to improve maintainability.

Here's a suggested approach:

def create_test_message(
    self,
    author: str,
    channel_id: str,
    channel_name: str,
    created_date: datetime,
    thread_id: str = None,
    thread_name: str = None,
) -> dict:
    """Create a test message with common structure."""
    return {
        "type": 0,
        "author": author,
        "content": f"test_message_{author}",
        "user_mentions": [],
        "role_mentions": [],
        "reactions": [],
        "replied_user": None,
        "createdDate": created_date,
        "messageId": f"msg_{author}_{created_date.strftime('%Y%m%d')}",
        "channelId": channel_id,
        "channelName": channel_name,
        "threadId": thread_id,
        "threadName": thread_name,
        "isGeneratedByWebhook": False,
    }

This would simplify the test data creation:

raw_data = [
    self.create_test_message(
        f"author_{i}",
        channels[i % len(channels)],
        "general",
        datetime(2023, 10, i + 1),
        thread_name="Something"
    )
    for i in range(2)
]

dags/hivemind_etl_helpers/tests/integration/test_github_get_communities_org.py (2)

Line range hint 76-78: Clean up or document commented test data fields

There are multiple commented-out fields in the test data (fromDate, repoIds, organizationId) across different test methods. These comments might indicate:

1. Upcoming features that are not yet implemented
2. Recently removed functionality
3. Changes in the data model

Please either:

• Remove these comments if they're no longer relevant
• Add a TODO comment explaining the future implementation plans
• Document why these fields are commented out

Also applies to: 164-166, 253-255, 306-308, 386-388, 434-436

---

Line range hint 332-333: Improve error message for better debugging

The error message could be more descriptive by including the actual unexpected organization IDs.

-                raise ValueError("No more organizations we had!")
+                raise ValueError(f"Unexpected organization_ids found: {res['organization_ids']}")

dags/analyzer_helper/tests/integration/test_integration_fetch_discord_platforms.py (2)

Line range hint 18-146: Consider refactoring test data setup to reduce duplication

The test data structure is repeated across multiple test methods. Consider extracting it to a helper method or fixture to improve maintainability.

Example refactor:

def create_sample_discord_platform(self, platform_id, guild_id, platform_name):
    return {
        "_id": ObjectId(platform_id),
        "name": "discord",
        "metadata": {
            "action": {
                "INT_THR": 1,
                # ... other thresholds
            },
            "window": {"period_size": 7, "step_size": 1},
            "id": guild_id,
            "isInProgress": False,
            "period": datetime(2023, 10, 20),
            "icon": "e160861192ed8c2a6fa65a8ab6ac337e",
            "selectedChannels": [
                "1067517728543477920",
                # ... other channels
            ],
            "name": platform_name,
            "analyzerStartedAt": datetime(2024, 4, 17, 13, 29, 16, 157000),
        },
        # ... other fields
    }

Also applies to: 147-275, 276-304, 305-319, 320-359, 360-479

---

Line range hint 305-319: Enhance error handling test with specific error message

The test for empty data in test_get_empty_data_fetch_analyzer_parameters could be more specific about the expected error message.

 def test_get_empty_data_fetch_analyzer_parameters(self):
     fetcher = FetchDiscordPlatforms()
     platform_id = ObjectId("000000000000000000000001")
-    # no results is given
-    with self.assertRaises(ValueError):
+    with self.assertRaisesRegex(ValueError, "No platform found with the given ID"):
         fetcher.fetch_analyzer_parameters(platform_id)

Configuration used: CodeRabbit UI
Review profile: CHILL

#!/bin/bash
# Description: Verify that all MongoSingleton imports are using the new path

# Check for any remaining old imports
echo "Checking for any remaining old imports..."
rg "from hivemind_etl_helpers.*mongo import MongoSingleton"

# Verify new imports are used consistently
echo "Verifying new imports..."
rg "from tc_hivemind_backend.db.mongo import MongoSingleton"

#!/bin/bash
# Check if these MongoDB operations are documented/used in the new backend
rg -A 2 -B 2 "delete_many|insert_many" --type py

# Check for any breaking changes or deprecation notices
rg -l "BREAKING.?CHANGES?|DEPRECATED" --type md

#!/bin/bash
# Check if tc_hivemind_backend is listed in requirements or setup files
rg -l "tc[_-]hivemind[_-]backend" *requirements*.txt setup.py pyproject.toml

#!/bin/bash
# Let's check the content of requirements.txt to see how tc_hivemind_backend is specified
cat requirements.txt | grep -A 2 -B 2 "tc[_-]hivemind[_-]backend"

# Also check if there are any other requirement files in the repository
fd "requirements.*\.txt$"

    def __init__(self, platform_id: str):
        super().__init__(platform_id)
-       # self._platform_id = platform_id
        self.client = MongoSingleton.get_instance().client
        self.db = self.client[self.get_platform_id()]
-       # self.db = self.client[self._platform_id]

#!/bin/bash
# Check if get_platform_id exists in the parent class
ast-grep --pattern 'class LoadTransformedMembersBase {
  $$$
  get_platform_id($$$) {
    $$$
  }
  $$$
}'

#!/bin/bash
# Let's try to find the base class file first
fd "LoadTransformedMembersBase" --type f

# Then let's search for get_platform_id method in all files
rg "get_platform_id" -A 3

def __init__(self, platform_id: str):
-    # super().__init__(platform_id)
+    super().__init__(platform_id)
    self._platform_id = platform_id
    self.client = MongoSingleton.get_instance().client
-    # self.db = self.client[self.get_platform_id()]
-    self.db = self.client[self._platform_id]
+    self.db = self.client[self.get_platform_id()]
    self.collection = self.db["rawmembers"]

def __init__(self, platform_id: str):
-    # super().__init__(platform_id)
+    super().__init__(platform_id)
-    self._platform_id = platform_id  # redundant as super().__init__() sets this
    self.client = MongoSingleton.get_instance().client
-    self.db = self.client[self._platform_id]
+    self.db = self.client[self.get_platform_id()]
    self.collection = self.db["rawmembers"]

#!/bin/bash
# Check if base class has essential initialization logic
ast-grep --pattern 'class LoadTransformedMembersBase {
  $$$
  __init__($$$) {
    $$$
  }
  $$$
}'

#!/bin/bash
# Let's first find the base class file
fd "LoadTransformedMembersBase" --type f

# Then let's search for any class definition containing LoadTransformedMembersBase
rg "class.*LoadTransformedMembersBase" -A 10

# Also search for any imports of this base class to understand the inheritance hierarchy
rg "from.*LoadTransformedMembersBase.*import" --type py

#!/bin/bash
# Check if tc_hivemind_backend is listed in requirements or setup files
rg -l "tc_hivemind_backend" "requirements*.txt" "setup*.py"

#!/bin/bash
# Let's first find any requirements or setup files in the repository
fd -t f "requirements|setup.py"

# Also check for any pyproject.toml or similar Python dependency files
fd -t f "pyproject.toml|poetry.lock"

# Let's also check if the package is imported elsewhere to understand its usage
rg -l "tc_hivemind_backend"

#!/bin/bash
# Let's check the content of requirements.txt for tc_hivemind_backend
cat requirements.txt

# Let's also check if this is a local package by looking for its directory
fd -t d "tc_hivemind_backend"

# Check for any setup.cfg or similar files that might contain dependency info
fd -t f "setup.cfg|deps.txt|dependencies"

#!/bin/bash
# Description: Verify that the new CustomIngestionPipeline class has the same interface
# Expected: Should find matching method signatures and required parameters

# Search for the old implementation
echo "Old implementation:"
rg -A 5 "class CustomIngestionPipeline" --glob "!target/"

# Search for the new implementation
echo "New implementation:"
ast-grep --pattern 'class CustomIngestionPipeline {
  $$$
  run_pipeline($$$) {
    $$$
  }
}'

#!/bin/bash
# Search for files containing CustomIngestionPipeline in both old and new locations
echo "Files with CustomIngestionPipeline:"
rg -l "CustomIngestionPipeline"

# Search for usage patterns of CustomIngestionPipeline
echo -e "\nUsage patterns:"
rg -B2 -A2 "CustomIngestionPipeline" 

# Look for the specific file that might contain the new implementation
echo -e "\nSearching tc_hivemind_backend directory:"
fd "ingest_qdrant.py"

#!/bin/bash
# Check if there are any other files that might contain the old implementation
echo "Searching for old implementation files:"
fd "ingestion_pipeline.py"

# Check imports in the test file for additional context
echo -e "\nChecking test file imports:"
head -n 20 dags/hivemind_etl_helpers/tests/integration/test_ingestion_pipeline_etl.py

#!/bin/bash
# Description: Check if the package requirements are specified in setup.py or requirements.txt
# and verify the minimum version needed for the new import path

# Check setup.py
echo "Checking setup.py for tc-hivemind-backend dependency..."
rg -l "tc[-_]hivemind[-_]backend" setup.py

# Check requirements files
echo "Checking requirements files..."
fd requirements.txt$ --exec rg -l "tc[-_]hivemind[-_]backend" {}

#!/bin/bash
# Let's check the actual content of requirements.txt to see the version specification
echo "Content of requirements.txt with tc-hivemind-backend:"
rg "tc[-_]hivemind[-_]backend" requirements.txt -A 1 -B 1

# Let's also check if there are any other files importing from tc_hivemind_backend.db.mongo
echo -e "\nFiles importing from tc_hivemind_backend.db.mongo:"
rg "from tc_hivemind_backend\.db\.mongo import" -l

# Check if there are any files still using the old import path
echo -e "\nFiles using old import path:"
rg "from hivemind_etl_helpers\.src\.utils\.mongo import" -l

#!/bin/bash
# Check if tc_hivemind_backend is properly configured in requirements
rg -l "tc[-_]hivemind[-_]backend" --type=txt

# Check for any remaining references to the old module
rg "hivemind_etl_helpers\.ingestion_pipeline" --type=python

#!/bin/bash
# Check the dependency configuration in requirements.txt
cat requirements.txt

# Check for any remaining references to the old module using ripgrep
# Note: Using --type-add to properly recognize Python files
rg --type-add 'py:*.py' --type py "hivemind_etl_helpers\.ingestion_pipeline"

# Check the new import usage
rg --type-add 'py:*.py' --type py "from tc_hivemind_backend.ingest_qdrant import"

# Look for the test file to verify its contents
fd "test_ingestion_pipeline_etl.py" --type f --exec cat {}

#!/bin/bash
# Description: Check if the CustomIngestionPipeline interface is compatible and if there are any version constraints

# Check for any requirements.txt or setup.py files that might specify the tc_hivemind_backend version
fd "requirements.*\.txt|setup\.py" | xargs rg "tc[-_]hivemind[-_]backend"

# Look for other usages of CustomIngestionPipeline to ensure consistent migration
rg -A 2 "CustomIngestionPipeline" --type python

# Check the class definition to verify the interface
ast-grep --pattern 'class CustomIngestionPipeline {
  $$$
  def run_pipeline($$$) {
    $$$
  }
}'

#!/bin/bash
# Let's try different commands to get more information

# Check for CustomIngestionPipeline usage with ripgrep (without file type)
rg -A 2 "CustomIngestionPipeline"

# Look for any migration guides or documentation
fd "README|MIGRATION" | xargs rg -i "hivemind|custom.*pipeline"

# Check the actual usage in the DAG file
cat dags/hivemind_google_drive_etl.py

# Check for any other files using the same import
rg "from tc_hivemind_backend.ingest_qdrant import"

#!/bin/bash
# Description: Check for any remaining relative imports of ModulesBase and verify consistent usage of the new absolute import

# Check for any remaining relative imports of ModulesBase
echo "Checking for remaining relative imports..."
rg "from \.*modules_base import ModulesBase"

# Check for the new absolute import pattern
echo "Checking new absolute import usage..."
rg "from tc_hivemind_backend\.db\.modules_base import ModulesBase"

#!/bin/bash
# Description: Compare MongoSingleton interface between old and new locations

# Check the new MongoSingleton implementation
echo "Checking new MongoSingleton implementation:"
ast-grep --pattern 'class MongoSingleton {
  $$$
}'

# Search for any remaining references to the old import path
echo "Checking for any remaining old import references:"
rg "from hivemind_etl_helpers.src.utils.mongo import MongoSingleton"

#!/bin/bash
# Let's try a different approach to verify the MongoSingleton usage and implementation

# First, let's check if we can find the new tc_hivemind_backend package
echo "Searching for tc_hivemind_backend package:"
fd "tc_hivemind_backend"

# Let's check for any Python files containing MongoSingleton
echo -e "\nSearching for MongoSingleton usage:"
rg "MongoSingleton" -t py

# Let's check the content of the current file to understand the usage
echo -e "\nChecking current file implementation:"
cat "dags/hivemind_etl_helpers/src/db/discord/utils/id_transform.py"

#!/bin/bash
# Description: Compare MongoSingleton interfaces between old and new packages

# Search for MongoSingleton class definitions and method signatures
echo "=== Searching for MongoSingleton class definitions ==="
ast-grep --pattern 'class MongoSingleton {
  $$$
}'

# Search for MongoSingleton usage patterns
echo -e "\n=== Searching for MongoSingleton usage patterns ==="
rg -A 2 'MongoSingleton\.get_instance\(\)'

#!/bin/bash
# Search for MongoSingleton class definition and its methods
echo "=== Searching for MongoSingleton class definition in Python files ==="
rg -t py "class MongoSingleton" -A 10

# Search for MongoSingleton imports to trace its source
echo -e "\n=== Searching for MongoSingleton imports ==="
rg -t py "^from.*MongoSingleton"

#!/bin/bash
# Description: Verify tc_hivemind_backend package setup and ModulesBase availability

# Check if the package is listed in requirements
rg -l "tc[-_]hivemind[-_]backend" requirements*.txt setup*.py

# Verify the module path exists and is importable
fd -e py "modules_base.py" | grep -i "tc_hivemind_backend/db"

#!/bin/bash
# Let's check the content of requirements.txt to see how tc_hivemind_backend is specified
cat requirements.txt | grep -i "tc[-_]hivemind[-_]backend"

# Let's also check if ModulesBase exists in the codebase
fd -e py "modules_base.py"

# Check the content of a found modules_base.py to verify the class
for f in $(fd -e py "modules_base.py"); do
    echo "=== Content of $f ==="
    cat "$f"
done

#!/bin/bash
# Description: Check for any remaining references to the old import path and verify new import usage

# Check for any remaining old import paths
echo "Checking for remaining old imports..."
rg "from hivemind_etl_helpers.src.utils.mongo import MongoSingleton"

# Check for consistent usage of new import
echo "Checking usage of new import path..."
rg "from tc_hivemind_backend.db.mongo import MongoSingleton"

# Verify no mixed usage of MongoSingleton
echo "Checking for any other MongoSingleton imports..."
rg "import.*MongoSingleton" --type py

#!/bin/bash
# Find all files using MongoSingleton to verify consistent usage
ast-grep --pattern 'from $_ import MongoSingleton'

# Check the method calls to ensure they match the expected interface
rg "MongoSingleton\.get_instance\(\)\.get_client\(\)"

#!/bin/bash
# Check if the tc_hivemind_backend package is properly specified in requirements
rg -l "tc[-_]hivemind[-_]backend" --type yaml --type txt --type=toml

# Check for any remaining old imports that need to be updated
rg "from hivemind_etl_helpers\.src\.utils\.mongo import MongoSingleton"

#!/bin/bash
# Check the actual content of requirements.txt to verify the backend library specification
cat requirements.txt

# Search for any MongoSingleton imports across the codebase to ensure consistency
rg "MongoSingleton" -A 2

#!/bin/bash
# Description: Check for any remaining relative imports of modules_base and verify the new import path usage

# Check for any remaining relative imports
echo "Checking for remaining relative imports of modules_base:"
rg "from \.*modules_base import"

# Verify the new import pattern usage
echo -e "\nVerifying new import pattern usage:"
rg "from tc_hivemind_backend\.db\.modules_base import"

#!/bin/bash
# Description: Compare the old and new CustomIngestionPipeline interfaces

# Search for the old implementation's interface
echo "Old Implementation:"
rg -A 10 "class CustomIngestionPipeline" --glob "!target/"

# Search for the new implementation's interface
echo "New Implementation:"
ast-grep --pattern 'class CustomIngestionPipeline {
  $$$
  $init($$$) {
    $$$
  }
  $$$
}'

#!/bin/bash
# Search for files containing CustomIngestionPipeline
echo "Files containing CustomIngestionPipeline:"
rg -l "CustomIngestionPipeline"

# Search for the old implementation in hivemind_etl_helpers
echo -e "\nOld implementation search:"
fd "ingestion_pipeline.py" --exec cat {}

# Search for the new implementation in tc_hivemind_backend
echo -e "\nNew implementation search:"
fd "ingest_qdrant.py" --exec cat {}

#!/bin/bash
# Look for CustomIngestionPipeline usage patterns in the files
echo "Usage in github_etl.py:"
rg -A 5 "CustomIngestionPipeline" dags/hivemind_etl_helpers/github_etl.py

echo -e "\nUsage in other ETL files:"
rg -A 5 "CustomIngestionPipeline" dags/hivemind_*.py dags/hivemind_etl_helpers/*.py

echo -e "\nUsage in tests:"
rg -A 5 "CustomIngestionPipeline" dags/hivemind_etl_helpers/tests/integration/test_ingestion_pipeline_etl.py

#!/bin/bash
# Description: Verify the tc_hivemind_backend package installation and MongoSingleton usage

# Check if the package is listed in requirements
rg -l "tc-hivemind-backend" "requirements*.txt" "setup*.py" "pyproject.toml"

# Verify MongoSingleton API consistency
ast-grep --pattern 'class MongoSingleton {
  $$$
  get_instance($$$) {
    $$$
  }
  $$$
}'

#!/bin/bash
# Let's try different approaches to verify the package and MongoDB usage

# Check for any requirements files with different naming patterns
fd -e txt -e pip requirements

# Look for package dependencies in any Python project files
fd -e toml -e yaml -e yml

# Search for MongoSingleton usage patterns in Python files
rg -l "MongoSingleton" -g "*.py"

# Look for the actual implementation of MongoSingleton in the new path
rg -p "class MongoSingleton" --type py

# Check if there are any other files still using the old import path
rg -l "hivemind_etl_helpers.src.utils.mongo" -g "*.py"

#!/bin/bash
# Let's check the requirements file for the backend package and examine the old vs new import usage

# Check requirements.txt content
cat requirements.txt

# Look for the new tc_hivemind_backend import pattern
rg -p "from tc_hivemind_backend.db.mongo import" --type py

# Check the content of test files still using old import to understand the migration status
rg -C 2 "hivemind_etl_helpers.src.utils.mongo" dags/violation_detection_helpers/tests/unit/test_extract_raw_data.py dags/violation_detection_helpers/tests/unit/test_extract_raw_data_latest_date.py

#!/bin/bash
# Description: Verify package installation and interface compatibility

# Check if the package is listed in requirements or setup files
echo "Checking package requirements..."
fd -e txt -e in requirements
fd setup.py | xargs cat

# Verify the interface compatibility by checking MongoSingleton usage
echo "Checking MongoSingleton interface..."
rg -A 5 "class MongoSingleton" 

#!/bin/bash
# Let's try a different approach to verify the package and interface

# Check the content of requirements.txt
echo "=== Content of requirements.txt ==="
cat requirements.txt

# Search for any references to tc_hivemind_backend
echo -e "\n=== References to tc_hivemind_backend ==="
rg "tc_hivemind_backend"

# Search for MongoSingleton usage patterns
echo -e "\n=== MongoSingleton usage patterns ==="
rg "MongoSingleton" -A 3

# Check if there are other similar mongo imports
echo -e "\n=== Other mongo imports ==="
rg "from.*mongo.*import"

#!/bin/bash
# Description: Verify the tc_hivemind_backend package installation and related imports

# Check if the package is listed in requirements
fd -t f "requirements.*txt" -x grep -l "tc-hivemind-backend"

# Check for any remaining old import patterns that might need updating
rg "from hivemind_etl_helpers.src.utils.mongo import" --type py

# Check for consistency in new import pattern usage
rg "from tc_hivemind_backend.db.mongo import" --type py

#!/bin/bash
# Search for MongoSingleton class definition and usage patterns
echo "Checking MongoSingleton interface in both old and new locations..."

# Check old implementation
echo "Old implementation:"
rg -A 10 "class MongoSingleton" 

# Check new implementation
echo "New implementation:"
ast-grep --pattern 'class MongoSingleton {
  $$$
  get_instance($$$) {
    $$$
  }
  $$$
  get_client($$$) {
    $$$
  }
}'

#!/bin/bash
# Search for MongoSingleton imports and usage
echo "Searching for MongoSingleton imports..."
rg "from.*MongoSingleton" -A 2

echo "\nSearching for MongoDB client usage patterns..."
rg "get_client\(\)" -A 2

echo "\nSearching for MongoDB operations..."
rg "\.(find_one|update_one|insert_one)\(" -A 2

#!/bin/bash
# Check if tc_hivemind_backend is listed in requirements or setup files
echo "Checking for tc_hivemind_backend in requirements and setup files..."
fd -e txt -e in requirements
fd setup.py | xargs cat

# Verify the new import path exists in the package
echo "Verifying the new import path exists..."
fd -t f -e py mongo.py | grep -r "tc_hivemind_backend/db"

#!/bin/bash
# Check requirements.txt content
echo "=== Checking requirements.txt content ==="
cat requirements.txt

# Search for any references to tc_hivemind_backend in Python files
echo -e "\n=== Searching for tc_hivemind_backend imports ==="
rg "from tc_hivemind_backend" -A 2

# Check if the old import path still exists
echo -e "\n=== Checking old import path ==="
rg "from hivemind_etl_helpers.src.utils.mongo"

# Look for MongoSingleton class definitions
echo -e "\n=== Looking for MongoSingleton class definitions ==="
ast-grep --pattern 'class MongoSingleton'

#!/bin/bash
# Look for related test files
echo "Checking for related test files..."
fd -t f -e py test_module.py

#!/bin/bash
# Check the interface of MongoSingleton in both locations
echo "Checking old MongoSingleton implementation..."
rg -A 10 "class MongoSingleton" 

# Check if there are any other files using the old import path
echo -e "\nChecking usage of old import path..."
rg "from tc_hivemind_backend.db.mongo import MongoSingleton"

# Check if there are any other files using the new import path
echo -e "\nChecking usage of new import path..."
rg "from hivemind_etl_helpers.src.db.telegram.utils import MongoSingleton"

#!/bin/bash
# Check the implementation in both locations
echo "Checking MongoSingleton implementation in tc_hivemind_backend..."
fd -t f mongo.py | xargs cat

echo -e "\nChecking MongoSingleton implementation in hivemind_etl_helpers..."
fd -t f module.py -p "hivemind_etl_helpers/src/db/telegram/utils" | xargs cat

Learnt from: amindadgar
PR: TogetherCrew/airflow-dags#221
File: dags/violation_detection_helpers/modules.py:7-8
Timestamp: 2024-11-12T14:59:34.318Z
Learning: The `MongoSingleton` class in `dags/hivemind_etl_helpers/src/utils/mongo.py` includes error handling within the `get_instance` method.

#!/bin/bash
# Description: Verify all MongoSingleton imports are updated consistently
# and no old imports remain

# Check for any remaining old imports
rg "from hivemind_etl_helpers.src.utils.mongo import MongoSingleton"

# Verify new imports are used consistently
rg "from tc_hivemind_backend.db.mongo import MongoSingleton"

#!/bin/bash
# Description: Verify all MongoSingleton imports have been updated consistently
# and no old imports remain

# Check for any remaining old imports
rg "from hivemind_etl_helpers.src.utils.mongo import MongoSingleton"

# Check for new imports to ensure consistency
rg "from tc_hivemind_backend.db.mongo import MongoSingleton"

#!/bin/bash
# Description: Check for schema definitions in the new backend library

# Look for schema definitions or models in the backend library
rg -A 10 "class.*Model|Schema" --type python

# Look for similar module/platform structures
ast-grep --pattern 'class $_ {
  $$$
  community: $_ 
  $$$
  platforms: $_
  $$$
}'

#!/bin/bash
# Let's try a different approach to find schema-related code

# Search for module-related schema or model definitions
rg -g '*.py' -A 10 'class.*Module|Schema.*module|model.*module' -i

# Search for platform configuration patterns
rg -g '*.py' -A 5 'platform.*metadata|platform.*config' -i

# Look for MongoDB schema validation or collection definitions
rg -g '*.py' -A 5 'collection.*modules|modules.*collection' -i

#!/bin/bash
# Let's check for tc_hivemind_backend imports and usage

# Search for tc_hivemind_backend imports
rg -g '*.py' -A 3 'from tc_hivemind_backend|import tc_hivemind_backend'

# Search for any references to hivemind backend schemas
rg -g '*.py' -A 5 'hivemind.*schema|schema.*hivemind'

# Look for discourse-specific platform configurations
rg -g '*.py' -A 5 'discourse.*config|config.*discourse' -i

#!/bin/bash
# Check if tc_hivemind_backend is declared in setup.py or requirements files
fd -e txt -e in requirements | xargs grep -l "tc_hivemind_backend"
fd "setup.py" | xargs grep -l "tc_hivemind_backend"

#!/bin/bash
# Let's first check what requirement files exist
fd -e txt -e in "requirements"

# Then let's check for setup.py files
fd "setup.py"

# Let's also check for pyproject.toml which is another common place for Python dependencies
fd "pyproject.toml"

# And check the content of any requirements files we find
fd -e txt -e in "requirements" -x cat {}

#!/bin/bash
# Check if the package is listed in requirements or setup files
rg -g "requirements*.txt" -g "setup.py" "tc_hivemind_backend"

# Verify the MongoSingleton interface compatibility
ast-grep --pattern 'class MongoSingleton {
  $$$
  get_instance($$$) {
    $$$
  }
  $$$
}'

#!/bin/bash
# Let's try a broader search for the MongoSingleton class and its usage
rg -l "MongoSingleton"

# Search for any Python requirements files in the repository
fd "requirements.*\.txt$|setup\.py$"

# Check the content of found requirement files
fd "requirements.*\.txt$|setup\.py$" -x cat {}

# Look for the actual implementation of MongoSingleton in both old and new paths
rg -A 5 "class MongoSingleton"

client = MongoSingleton.get_instance().get_client()

#!/bin/bash
# Search for MongoSingleton class definition to verify method existence
ast-grep --pattern 'class MongoSingleton {
  $$$
  get_instance($$$) {
    $$$
  }
  $$$
  get_client($$$) {
    $$$
  }
  $$$
}'

#!/bin/bash
# Let's try a broader search for MongoSingleton class and its methods
rg -A 10 "class MongoSingleton"

# Also search for any imports or usage of MongoSingleton
rg "MongoSingleton"

# Search specifically for get_instance and get_client methods
rg "get_instance|get_client"

#!/bin/bash
# Check if tc_hivemind_backend is declared as a dependency
fd "setup.py|requirements.txt|pyproject.toml" --exec grep -l "tc_hivemind_backend"

#!/bin/bash
# Let's first find all dependency files
fd "setup.py|requirements.txt|pyproject.toml"

#!/bin/bash
# Let's also check if the package is installed via git dependency
fd "setup.py|requirements.txt|pyproject.toml" --exec grep -l "git+"

#!/bin/bash
# Let's check the content of these files to see any indirect dependencies
fd "setup.py|requirements.txt|pyproject.toml" --exec cat {}

#!/bin/bash
# Let's also check if there are any other files importing this package
rg "from tc_hivemind_backend|import tc_hivemind_backend" -l

-    client = MongoSingleton.get_instance().client
+    client = MongoSingleton.get_instance().get_client()

-    client = MongoSingleton.get_instance().client
+    client = MongoSingleton.get_instance().get_client()

#!/bin/bash
# Search for Core database usage patterns
rg -l "Core.*platforms|Core.*modules" --type py

#!/bin/bash
# Check test files for Core database structure
rg -A 10 "Core.*platforms|Core.*modules" dags/hivemind_etl_helpers/tests/integration/test_discord_fetch_raw_messages.py dags/hivemind_etl_helpers/tests/integration/test_discord_fetch_modules_channels.py

#!/bin/bash
# Look for migration scripts or schema changes related to Core database
rg -g "*.py" -g "!*test*" "Core.*platforms|Core.*modules" dags/migrations/ dags/schemas/

#!/bin/bash
# Check if tc_hivemind_backend is listed in setup.py or requirements files
rg "tc_hivemind_backend" | grep -E "setup.py|requirements"

#!/bin/bash
# Let's first check if the package exists in any requirements or setup files
fd "requirements|setup.py" --type f --exec cat {}

# Also check if the package is being used elsewhere in the codebase
rg "tc_hivemind_backend" -A 2

#!/bin/bash
# Description: Check for any remaining old import paths and verify new import usage

# Check for any remaining old imports
echo "Checking for any remaining old imports..."
rg "from hivemind_etl_helpers.ingestion_pipeline import CustomIngestionPipeline"

# Verify the new import is used consistently
echo "Verifying new import usage..."
rg "from tc_hivemind_backend.ingest_qdrant import CustomIngestionPipeline"

# Check for any direct usage of the old module
echo "Checking for any direct usage of old module..."
rg "hivemind_etl_helpers.ingestion_pipeline"

#!/bin/bash
# Check if tc_hivemind_backend is listed in requirements files or setup.py
echo "Checking for tc_hivemind_backend in requirements and setup files..."
fd -e txt -e in requirements
fd setup.py | xargs cat | grep -i "tc_hivemind_backend"

#!/bin/bash
# Let's check the content of requirements.txt and search for any MongoDB-related dependencies
echo "=== Content of requirements.txt ==="
cat requirements.txt

# Let's also check if there are any other requirement files in subdirectories
echo -e "\n=== Other requirement files ==="
fd requirements.txt -x echo "=== {} ===" \; -x cat {}

# Let's also check for any MongoDB imports in the test file
echo -e "\n=== MongoDB imports in the test file ==="
cat "dags/hivemind_etl_helpers/tests/integration/test_discord_prepare_llama.py"

#!/bin/bash
# Description: Check for any remaining old import paths and verify new import paths

# Check for any remaining old imports
echo "Checking for old import paths..."
rg "from hivemind_etl_helpers.src.utils.mongo import MongoSingleton"

# Check new import usage
echo "Checking new import paths..."
rg "from tc_hivemind_backend.db.mongo import MongoSingleton"

#!/bin/bash
# Description: Verify consistent usage of the new MongoDB import across the codebase
# and check for any potential breaking changes or inconsistencies

# Check for any remaining old import paths
echo "Checking for any remaining old imports..."
rg "from hivemind_etl_helpers.src.utils.mongo import MongoSingleton"

# Verify consistent usage of new import
echo "Verifying consistent usage of new import..."
rg "from tc_hivemind_backend.db.mongo import MongoSingleton"

# Look for any direct mongo imports that might need updating
echo "Checking for other mongo-related imports that might need updating..."
rg -g '!*.pyc' -g '!*.git/*' "import.*mongo"

#!/bin/bash
# Description: Check for consistent MongoDB client initialization patterns
# and potential breaking changes in connection management

# Look for all MongoSingleton.get_instance() usage patterns
echo "Checking MongoSingleton usage patterns..."
ast-grep --pattern 'MongoSingleton.get_instance()'

# Check for any custom MongoDB configuration or setup
echo "Checking for MongoDB configuration patterns..."
rg -g '*.py' "MongoSingleton.*configure|MongoSingleton.*setup|MongoSingleton.*init"

#!/bin/bash
# Let's check the MongoSingleton implementation and initialization patterns

# Find MongoSingleton class definition and implementation
echo "Checking MongoSingleton implementation..."
ast-grep --pattern 'class MongoSingleton'

# Look for MongoDB client initialization in the codebase
echo "Checking MongoDB client initialization patterns..."
rg -g '*.py' "from pymongo import MongoClient"

# Check for any direct MongoDB client instantiation
echo "Checking direct MongoDB client usage..."
rg -g '*.py' "MongoClient\(" -A 2

#!/bin/bash
# Let's try a different approach to find MongoDB related code

# Search for MongoDB singleton pattern in Python files
echo "Searching for MongoDB singleton files..."
fd -e py -x grep -l "Mongo" {} \;

# Look for MongoDB related imports and configurations
echo "Checking MongoDB related imports..."
rg -g '*.py' "import.*[Mm]ongo|from.*[Mm]ongo"

# Check for MongoDB client property or method definitions
echo "Checking MongoDB client definitions..."
ast-grep --pattern 'client = $_'

#!/bin/bash
# Check if tc_hivemind_backend is listed in requirements files or setup.py
echo "Searching for tc_hivemind_backend in requirements and setup files..."
fd -e txt -e in requirements setup.py | xargs rg "tc_hivemind_backend"

#!/bin/bash
# Check if tc_hivemind_backend is listed in requirements files or setup.py
fd -e txt -e in "requirements" -x cat {} \; | grep -i "tc[-_]hivemind[-_]backend"
fd "setup.py" -x cat {} \; | grep -i "tc[-_]hivemind[-_]backend"

#!/bin/bash
# Description: Verify the MongoSingleton interface in both old and new implementations

# Check the old implementation
rg -A 5 "class MongoSingleton" 

# Check method signatures in the new implementation
ast-grep --pattern 'class MongoSingleton {
  $$$
  get_instance() {
    $$$
  }
  $$$
  get_client() {
    $$$
  }
  $$$
}'

#!/bin/bash
# Let's try a broader search for MongoSingleton implementations
# First, let's find all files that might contain MongoSingleton
rg -l "MongoSingleton"

# Then, let's look for the class definition and its methods with more context
rg -A 10 "class MongoSingleton"

# Also search for any imports of MongoSingleton to understand usage patterns
rg "from.*MongoSingleton|import.*MongoSingleton"

#!/bin/bash
# Let's check the content of the test file to understand how MongoSingleton is used
cat "dags/violation_detection_helpers/tests/integration/test_extract_data.py"

# Also check one of the source files to see the actual usage pattern
cat "dags/violation_detection_helpers/extract.py"

# Let's also check if we can find the actual implementation in tc_hivemind_backend
rg -g "*.py" -A 10 "class MongoSingleton" --no-ignore

#!/bin/bash
# Description: Compare the MongoSingleton interface between old and new implementations

# Search for MongoSingleton class definition in both old and new locations
echo "=== Old MongoSingleton implementation ==="
rg -A 10 "class MongoSingleton" hivemind_etl_helpers/src/utils/mongo.py || echo "Old file not found"

echo -e "\n=== New MongoSingleton implementation ==="
rg -A 10 "class MongoSingleton" tc_hivemind_backend/db/mongo.py || echo "New file not found"

# Search for all MongoSingleton method calls to ensure we're not missing any usage patterns
echo -e "\n=== MongoSingleton usage patterns ==="
rg "MongoSingleton\." --type py

#!/bin/bash
# Search for MongoSingleton class definition across the codebase
echo "=== MongoSingleton class definition ==="
fd -e py | xargs ast-grep --pattern 'class MongoSingleton'

# Get the content of files containing MongoSingleton class
echo -e "\n=== MongoSingleton implementation details ==="
rg -B 2 -A 15 "class MongoSingleton"

#!/bin/bash
# Description: Verify MongoSingleton API compatibility between old and new implementations

# Check for any differences in MongoSingleton class definition
echo "Checking MongoSingleton class definition..."
rg -A 10 "class MongoSingleton" 

# Look for any remaining references to the old import path
echo "Checking for any remaining old import paths..."
rg "from hivemind_etl_helpers.src.utils.mongo import MongoSingleton"

#!/bin/bash
# Let's try a different approach to verify the MongoSingleton implementation

# First, let's find all files that might contain MongoSingleton
echo "Files containing MongoSingleton:"
rg -l "MongoSingleton"

# Check all imports of MongoSingleton to ensure consistency
echo -e "\nAll MongoSingleton imports:"
rg "import.*MongoSingleton"

# Look for the actual implementation in tc_hivemind_backend
echo -e "\nSearching for MongoSingleton implementation:"
fd -e py -e js -e ts . | xargs rg "class MongoSingleton" -A 5

# Check usage patterns in the test file
echo -e "\nUsage in the test file:"
rg "MongoSingleton" "dags/hivemind_etl_helpers/tests/integration/test_discord_fetch_raw_messages_grouped.py" -A 2

#!/bin/bash
# Description: Verify all MongoSingleton imports have been updated consistently

# Check for any remaining old imports
echo "Checking for any remaining old imports..."
rg "from hivemind_etl_helpers.src.utils.mongo import MongoSingleton"

# Verify new imports
echo "Verifying new imports..."
rg "from tc_hivemind_backend.db.mongo import MongoSingleton"

#!/bin/bash
# Description: Verify consistent usage of the new MongoSingleton import across the codebase
# Expected: All MongoSingleton imports should be from tc_hivemind_backend.db.mongo

# Search for any remaining old imports
rg "from hivemind_etl_helpers.src.utils.mongo import MongoSingleton"

# Search for new imports to verify consistency
rg "from tc_hivemind_backend.db.mongo import MongoSingleton"

#!/bin/bash
# Description: Verify MongoDB collection structure alignment
# Expected: Collection names and schema should match tc_hivemind_backend expectations

# Search for collection definitions in the backend library
rg -A 5 "Core\"\]\[\"(modules|platforms|tokens)\"\]" --type py

# Search for any schema definitions in the backend library
ast-grep --pattern 'class $name {
  $$$
  collection = "Core"
  $$$
}'

#!/bin/bash
# Search for ModulesGDrive class implementation and its dependencies
rg -g '!*test*' -A 10 "class ModulesGDrive" --type py

# Search for MongoDB schema/model definitions in the codebase
rg -g '!*test*' "class.*Schema.*Core" --type py

#!/bin/bash
# Search for ModulesBase class implementation
rg -g '!*test*' -A 15 "class ModulesBase" --type py

# Search for MongoDB collection access patterns in the base class
rg -g '!*test*' "self\.client\[\"Core\"\]" --type py

#!/bin/bash
# Get the full implementation of ModulesGDrive class
cat dags/hivemind_etl_helpers/src/utils/modules/gdrive.py

# Search for any MongoDB schema definitions in the codebase that might be related to GDrive
rg -g '!*test*' -A 5 "mongodb.*google" --type py
rg -g '!*test*' -A 5 "collection.*google" --type py

#!/bin/bash
# Search for any remaining old import paths
rg "from hivemind_etl_helpers.src.utils.mongo import MongoSingleton"

# Verify the new import path usage
rg "from tc_hivemind_backend.db.mongo import MongoSingleton"

#!/bin/bash
# Description: Verify that all imports of MongoSingleton use the new path

# Check for any remaining old imports
echo "Checking for old imports..."
rg "from hivemind_etl_helpers.*import.*MongoSingleton"

# Verify all new imports are consistent
echo "Verifying new imports..."
rg "from tc_hivemind_backend\.db\.mongo import MongoSingleton"